What

Adds an optional expectedOutput field at the evaluation item level in the v1.0 schema. Output-based evaluators (ExactMatch, JsonSimilarity, LLMJudgeOutput) can now read expected output from the evaluation itself, instead of requiring it to be duplicated in every evaluator's evaluationCriterias entry.

Why

Today, when three evaluators need the same expected output, you write it three times:

"evaluationCriterias": {
  "exact-match":       { "expectedOutput": { "result": 4 } },
  "json-similarity":   { "expectedOutput": { "result": 4 } },
  "llm-judge-output":  { "expectedOutput": { "result": 4 } }
}

With this change you write it once:

"expectedOutput": { "result": 4 },
"evaluationCriterias": {
  "exact-match": null,
  "json-similarity": null,
  "llm-judge-output": null
}

Per-evaluator override still works — if a criteria entry has its own expectedOutput, it wins.

How it works

Model — One new optional field on EvaluationItem:

expected_output: dict[str, Any] | str | None = Field(default=None, alias="expectedOutput")

Runtime — 15 lines of merge logic in _execute_eval() before criteria is passed to evaluators:

1. If the evaluator's criteria type extends OutputEvaluationCriteria and the evaluation has expectedOutput:
   • Null criteria → inject {"expectedOutput": eval_item.expected_output}
   • Criteria without expectedOutput → merge it in
   • Criteria with expectedOutput → keep as-is (per-evaluator wins)
2. Non-output evaluators (Contains, ToolCall*, Trajectory) are completely untouched.

Backward compatibility

• The new field defaults to None — existing eval-set JSONs parse and run without any changes.
• Legacy evaluation sets are unaffected (different model, different migration path).
• No changes needed in downstream repos (uipath-agents, uipath-langchain, uipath-runtime).

Tests

25 new tests across 7 test classes covering:

• Model parsing (dict, string, null, missing, serialization roundtrip)
• Runtime merge logic (null criteria, override, injection, non-output evaluators, edge cases)
• Evaluator integration (ExactMatch, JsonSimilarity, LLMJudgeOutput)
• Legacy migration compatibility
• End-to-end flow simulation (mixed evaluator types, override precedence)

All 1574 tests pass (25 new + 1549 existing, zero regressions).

Jira

AE-1066

Spec

Confluence: Evaluation-Level ExpectedOutput Schema Enhancement

🤖 Generated with Claude Code